{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compare accuracy of four CNN architectures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we compare accuracy of four CNN architectures:\n",
    "\n",
    "- [AlexNet](https://github.com/BVLC/caffe/tree/master/models/bvlc_alexnet)\n",
    "- [SqueezeNet 1.0](https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.0)\n",
    "- [SqueezeNet 1.1](https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.1)\n",
    "- [GoogleNet](https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet)\n",
    "\n",
    "on the [Imagenet validation set](http://academictorrents.com/details/5d6d0df7ed81efd49ca99ea4737e0ae5e3a5f2e5) (50,000 images)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experimental data is stored in the main CK-Caffe repository under '[experiment](https://github.com/dividiti/ck-caffe/tree/master/experiment)'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Includes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collective Knowledge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import ck.kernel as ck\n",
    "print ('CK version: %s' % ck.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scientific"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import IPython as ip\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import pandas as pd\n",
    "import matplotlib as mp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print ('IPython version: %s' % ip.__version__)\n",
    "print ('NumPy version: %s' % np.__version__)\n",
    "print ('SciPy version: %s' % sp.__version__)\n",
    "print ('Pandas version: %s' % pd.__version__)\n",
    "print ('Matplotlib version: %s' % mp.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from matplotlib import cm\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Access experimental results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def get_run_info(tags):\n",
    "    r=ck.access({'action':'get', 'module_uoa':'experiment', 'tags':tags, 'load_json_files':['0001']})\n",
    "    if r['return']>0:\n",
    "        print (\"Error: %s\" % r['error'])\n",
    "        exit(1)\n",
    "    # Assume a single point per the given tags.\n",
    "    point = r['points'][0]\n",
    "    characteristics = point['0001']['characteristics_list']\n",
    "    run_info = characteristics[0]['run']\n",
    "    return run_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AlexNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "alexnet_tags = 'accuracy,caffemodel,bvlc,alexnet'\n",
    "alexnet_info = get_run_info(alexnet_tags)\n",
    "alexnet_accuracy = (alexnet_info['accuracy'], alexnet_info['accuracy_top5'])\n",
    "alexnet_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SqueezeNet 1.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "squeezenet_1_0_tags = 'accuracy,caffemodel,deepscale,squeezenet-1.0'\n",
    "squeezenet_1_0_info = get_run_info(squeezenet_1_0_tags)\n",
    "squeezenet_1_0_accuracy = (squeezenet_1_0_info['accuracy'], squeezenet_1_0_info['accuracy_top5'])\n",
    "squeezenet_1_0_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SqueezeNet 1.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "squeezenet_1_1_tags = 'accuracy,caffemodel,deepscale,squeezenet-1.1'\n",
    "squeezenet_1_1_info = get_run_info(squeezenet_1_1_tags)\n",
    "squeezenet_1_1_accuracy = (squeezenet_1_1_info['accuracy'], squeezenet_1_1_info['accuracy_top5'])\n",
    "squeezenet_1_1_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GoogleNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "googlenet_tags = 'accuracy,caffemodel,bvlc,googlenet'\n",
    "googlenet_info = get_run_info(googlenet_tags)\n",
    "googlenet_accuracy = (googlenet_info['loss3/top-1'], googlenet_info['loss3/top-5'])\n",
    "googlenet_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(\n",
    "    columns=[['Accuracy, %']*2, ['Top 1', 'Top 5']],\n",
    "    data=[alexnet_accuracy, squeezenet_1_0_accuracy, squeezenet_1_1_accuracy, googlenet_accuracy], \n",
    "    index=['AlexNet', 'SqueezeNet 1.0', 'SqueezeNet 1.1', 'GoogleNet']\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['Accuracy, %'].T \\\n",
    "    .plot(title='Prediction accuracy on the ImageNet validation set (50,000 images)',\n",
    "          kind='bar', rot=0, ylim=[0,1], figsize=[20, 8], grid=True, legend=True, colormap=cm.autumn, fontsize=16)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
